Active parallax correction

ABSTRACT

Techniques of active parallax correction are disclosed. In some embodiments, a first gaze direction of at least one eye of a user is determined. A determination about virtual content can then be made based on the first gaze direction, and the virtual content can be caused to be presented to the user based on the determination. In some embodiments, making the determination comprises determining a first location on a display surface at which to display the virtual content. In some embodiments, the virtual content can be caused to be displayed on the display surface at the first location.

TECHNICAL FIELD

The present application relates generally to the technical field of dataprocessing, and, in various embodiments, to methods and systems ofactive parallax correction.

BACKGROUND

Heads-ups display devices allow users to observe a scene whilesimultaneously seeing relevant virtual content that may be aligned(beneficially) to item, images, objects, or environments in the field ofview of the device or user. However, existing heads-up display devicesare unable to accurately project rendered computer graphics onto thehuman retina so that it is registered with the scene as the eye rotatesin the orbit.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the present disclosure are illustrated by way ofexample and not limitation in the figures of the accompanying drawings,in which like reference numbers indicate similar elements, and in which:

FIG. 1 is a block diagram illustrating a computing device, in accordancewith some embodiments;

FIG. 2 is a block diagram illustrating a virtual content module, inaccordance with some embodiments;

FIGS. 3A-3B illustrate an adjustment of a display location of virtualcontent based on a change in a gaze direction of a user, in accordancewith some embodiments;

FIG. 4 illustrates a heads-up display device incorporating activeparallax features, in accordance with some embodiments;

FIGS. 5A-5B illustrate parameters that can be used in employing activeparallax correction in determining a display location of virtualcontent, in accordance with some embodiments;

FIG. 6 is a flowchart illustrating a method of active parallaxcorrection, in accordance with some embodiments;

FIG. 7 is a flowchart illustrating a method of active parallaxcorrection, in accordance with some embodiments;

FIG. 8 is a flowchart illustrating a method of using a gaze direction inthe presentation of virtual content to a user, in accordance with someembodiments;

FIG. 9 is a block diagram of an example computer system on whichmethodologies described herein may be executed, in accordance with someembodiments; and

FIG. 10 is a block diagram illustrating a mobile device, in accordancewith some embodiments.

DETAILED DESCRIPTION

Example methods and systems of active parallax correction are disclosed.In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of example embodiments. It will be evident, however, toone skilled in the art that the present embodiments may be practicedwithout these specific details.

The present disclosure provides techniques for adjusting the displaylocation of virtual content on a display surface based on a detectedshift in the gaze direction of a user. In some embodiments, the cornealcenter of the user's eye(s) is estimated and used for the purpose ofadjusting the projected light, which forms the virtual content,reflected off a display surface.

In some embodiments, a first gaze direction of at least one eye of auser is determined. A determination about virtual content can then bemade based on the first gaze direction, and the virtual content can becaused to be presented to the user based on the determination.

In some embodiments, making the determination comprises determining afirst location on a display surface at which to display the virtualcontent. In some embodiments, causing the virtual content to be presentto the user comprises causing the virtual content to be displayed on thedisplay surface at the first location.

In some embodiments, a second gaze direction of the at least one eye ofthe user is determined, where the second gaze direction is differentfrom the first gaze direction. A second location on the display surfaceat which to display the virtual content can then be determined based onthe second gaze direction, where the second location is different fromthe first location. The virtual content can then be caused to bedisplayed on the display surface at the second location.

In some embodiments, image data of real-world visual content isreceived. The image data can be image data that has been captured by animage capture device. The virtual content can be determined based on thereceived image data. In some embodiments, the determination of the firstlocation is further based on a distance between the at least one eye andthe display surface and a distance between the at least one eye and thereal-world visual content. In some embodiments, determining the firstlocation comprises: emitting at least one light ray from at least onelight emitting diode towards the at least one eye; observing at leastone glint reflected off of the at least one eye using at least onecamera; determining a corneal center of the at least one eye based onthe observed at least one glint; determining a path of incident lightfrom the real-world visual content to the corneal center of the at leastone eye; and adjusting positioning of at least one virtual content lightray forming the virtual content on the display surface to make at leastone reflected light ray reflecting off of the display surface as aresult of the at least one virtual content light ray coincident with theincident light ray from the real-world visual content.

In some embodiments, the display surface is transparent. In someembodiments, the display surface and the image capture device arecoupled to a heads-up display device. In some embodiments, the heads-updisplay device comprises an optical head-mounted display device.

The methods or embodiments disclosed herein may be implemented as acomputer system having one or more modules (e.g., hardware modules orsoftware modules). Such modules may be executed by one or moreprocessors of the computer system. The methods or embodiments disclosedherein may be embodied as instructions stored on a machine-readablemedium that, when executed by one or more processors, cause the one ormore processors to perform the instructions.

FIG. 1 is a block diagram illustrating a computing device 100, inaccordance with some embodiments. Computing device 100 may comprise asmart phone, a tablet computer, a wearable computing device, a vehiclecomputing device, a laptop computer, and a desktop computer. In someembodiments, computing device 100 comprises a heads-up display device. Aheads-up display device is a device that provides a transparent displayscreen that presents content without requiring the user to look awayfrom his or her usual viewpoint, such as with the user's head positionedup and looking forward, instead of angled down to look at a device. Insome embodiments, the heads-up display device comprises an opticalhead-mounted display device, which may include, but is not limited to, ahelmet mounted display device, glasses (e.g., Google Glass®), or othertemporary or permanent form factors that can be either binocular ormonocular. However, it is contemplated that other types of computingdevices 100 are also within the scope of the present disclosure. In someembodiments, the computing device 100 comprises one or more imagecapture devices 110, a display screen 120, memory 130, and one or moreprocessors 140.

In some embodiments, the image capture device(s) 110 comprises abuilt-in camera or camcorder with which a user of the computing device100 can use to capture image data of visual content in a real-worldenvironment (e.g., a real-world physical object). The image data maycomprise one or more still images or video. As will be discussed infurther detail herein, the image capture device(s) 110 can also be usedto capture image data related to the positioning or orientation of atleast one of the user's eyes, such as a gaze direction of the user'seyes (e.g., where the user is looking or the rotational position of theuser's eyes relative to the user's head or some other point ofreference). Other configurations of the image capture device(s) 110 arealso within the scope of the present disclosure.

In some embodiments, the display screen 120 is configured to display theimage data captured by the image capture device 110. In someembodiments, the display screen 120 is transparent or semi-opaque sothat the user of the computing device 100 can see through the displayscreen 120 to the visual content in the real-world environment.

In some embodiments, a virtual content module 150 is stored in memory130 or implemented as part of the hardware of the processor(s) 140, andis executable by the processor(s) 140. Although not shown, in someembodiments, the virtual content module 150 may reside on a remoteserver and communicate with the computing device 100 via a network. Thenetwork may be any network that enables communication between or amongmachines, databases, and devices. Accordingly, the network may be awired network, a wireless network (e.g., a mobile or cellular network),or any suitable combination thereof. The network may include one or moreportions that constitute a private network, a public network (e.g., theInternet), or any suitable combination thereof.

FIG. 2 is a block diagram illustrating components of virtual contentmodule 150, in accordance with some embodiments. In some embodiments,virtual content module 150 comprises a virtual content determinationmodule 210, an eye tracking module 220, and an active parallaxcorrection module 230. Other configurations are also within the scope ofthe present disclosure.

In some embodiments, virtual content determination module 210 isconfigured to determine virtual content to be presented to a user. Insome embodiments, the virtual content comprises visual content, such ascomputer graphics (e.g., still images, video, animation, etc.).Accordingly, the presentation of the virtual content can comprise thevirtual content being displayed on display screen 120. The virtualcontent can be selected, or otherwise determined, from one or more setsof virtual content stored on the computing device 100 and/or from one ormore sets of virtual content stored on one or more remote servers. Insome embodiments, the virtual content module 150 is configured toprovide an augmented reality experience over real-world visual content.Augmented reality provides a live, direct or indirect, view of aphysical, real-world environment whose elements are augmented bycomputer-generated sensory input such as sound, video, graphics or GPSdata. In some embodiments, the virtual content determination module 210is configured to receive image data of real-world visual content (e.g.,a real-world physical object), which may have been captured using imagecapture device(s) 110, and determine virtual content to be displayed ondisplay screen 120 based on the image data of the real-world visualcontent.

In some embodiments, the virtual content comprises non-visual content.The virtual content can comprise audio content (e.g., music, speech,etc.). In some embodiments, the virtual content can comprise hapticfeedback (e.g., applying forces, vibrations, motions, etc.).

In some embodiments, eye tracking module 220 is configured to determinea gaze direction of at least one eye of a user of the computing device100. It is contemplated that eye tracking module 220 may employ anysuitable eye tracking techniques, including, but not limited to, opticaltracking techniques. In some embodiments, one or more image captureddevices, such as image capture device(s) 110 can be used to captureimage data used in any eye tracking techniques employed by the eyetracking module 220.

In some embodiments, active parallax correction module 230 is configuredto determine a location on a display surface of the display screen 120at which to display the virtual content. The active parallax correctionmodule 230 can determine this location based on the determined gazedirection of the user's eye(s). In some embodiments, determining thisdisplay location comprises making an adjustment to an initial displaylocation. For example, an initial display location for the virtualcontent on the display surface of the display screen may be initiallydetermined. This initial display location can be determined based on adetermined positioning of the computing device 100. For example thepositioning of an optical head-mounted display device being used by theuser relative to real-world visual content can be determined based ondata captured by one or more image capture devices on the head-mounteddisplay device, such as image data of the real-world visual content. Anadjustment may then be made to this initial display location based onthe determined gaze direction of the user, thereby resulting in a newlydetermined display location on the display surface at which to displaythe virtual content. Such an adjustment may comprise a shifting of thedisplay of the virtual content from the initial display location to thenewly determined display location. The active parallax correction module230 can be further configured to cause the virtual content to bedisplayed at the determined location on the display surface of thedisplay screen 120. In this respect, the active parallax correctionmodule 230 can adjust the display location of virtual content based on achange in the gaze direction of the user in order to maintain fidelityfrom the user's changed perspective, thereby using the gaze direction asa refinement mechanism for the initially determined display location forthe virtual content.

FIGS. 3A-3B illustrate an adjustment of a display location of virtualcontent based on a change in a gaze direction of a user, in accordancewith some embodiments. FIG. 3A illustrates at least one of the user'seyes 310 looking through a display surface 330 at real-world visualcontent 320 along a first gaze direction 315. Virtual content 335 iscaused to be displayed on display surface 330 at a first location basedon the first gaze direction 315. As seen in FIG. 3B, the user's gazedirection may shift from the first gaze direction 315 to a second gazedirection 315′. Accordingly, the location at which virtual content 335is caused to be displayed on display surface 330 can be shifted from thefirst location to a second location, as shown in FIG. 3B with theshifting of virtual content 335 from the first location to virtualcontent 335′ at the second location. In some embodiments, virtualcontent 335′ is identical to virtual content 335 (e.g., the same visualobject with the same properties). In some embodiments, virtual content335′ is a modified version of virtual content 335 (e.g., the same visualobject with different properties). In some embodiments, virtual content335′ is completely different from virtual content 335 (e.g., a differentvisual object).

Display surface 330 may be part of display screen 120 of computingdevice 100. In some embodiments, display surface 330 is curved, as shownin FIGS. 3A-3B. In some embodiments, display surface 330 is straight orflat (not shown). In some embodiments, display surface 330 may be onecontinuous display surface. In some embodiments, display surface 330 maybe formed from two or more distinct display surfaces configured to worktogether. It is contemplated that other shapes and configurations ofdisplay surface 330 are also within the scope of the present disclosure.

FIG. 4 illustrates a heads-up display device 400 incorporating activeparallax features disclosed herein, in accordance with some embodiments.In some embodiments, heads-up display device 400 comprises a deviceframe 440 to which its components may be coupled and via which the usercan mount, or otherwise secure, the heads-up display device 400 on theuser's head 405. Although device frame 400 is shown in FIG. 4 having arectangular shape, it is contemplated that other shapes of device frame440 are also within the scope of the present disclosure.

The user's eyes 410 a and 410 b can look through a display surface 330of the heads-up display device 400 at real-world visual content 320.Each eye has a corneal center 412 and a pupil 414. In order to make FIG.4 easier to view, certain operations are shown as being performed withrespect to eye 410 a, but not eye 410 b. However, correspondingoperations can be performed with respect to eye 410 b as well, eitherseparately or in conjunction with eye 410 a. In this fashion, eyes 410 aand 410 b can be treated separately on their own or together as a singlecombined unit.

In some embodiments, heads-up display device 400 comprises a stereocamera system comprising cameras 460 a and 460 b directed towards theuser's eyes 410 a and 410 b, respectively. In other embodiments, onecentral eye tracking camera (not shown) may be employed. Each camera canbe equipped with near-infrared band-pass filters, and can have paraxialnear-infrared light emitting diodes 462. In some embodiments, the lightemitting diodes 462 are placed near the optical centers of theirrespective cameras 460 a and 460 b. The cameras 460 a and 460 b can havenear-infrared band-pass filters to capture the light from theirrespective near-infrared light emitting diodes 462.

In some embodiments, heads-up display device 400 also comprises twodistal projectors 450 a and 450 b configured to emit light towardsdisplay surface 330. Display surface 330 can be configured to provideoptical see-through (transparent) ability and reflectivity propertiesfor the distal light projectors 450 a and 450 b. It is contemplated thatother types, numbers, and configurations of light emitting projectorscan also be employed and are within the scope of the present disclosure.

In FIG. 4, a stereo eye tracking system is being employed to estimatethe corneal center 412 of eyes 410 a and 410 b. The eyes 410 a and 410 bcan be modeled as two pinhole cameras. The estimated corneal center 412of each eye 410 a, 410 b can then be used to direct light rays from thecorresponding projector 450 a, 450 b so that emitted ray 452 reflectsoff display surface 330 such that the reflected ray 454 is coincidentwith the incident light ray 456 from real-world visual content 320, asshown in FIG. 4 with reflected ray 454 and light ray 454 being alignedalong the same dotted line. Intersection point 470 represents the pointof intersection of the light ray from visual content 320 with displaysurface 330.

In some embodiments, the near-infrared light emitting diodes 462 emitapproximately paraxial light rays to the observing cameras 460 a and 460b. The corneal eye surface of each eye 410 a and 410 b can be modeled bythe eye tracking module 220 as a convex mirror. Convex mirrors have theproperty that an incident ray directed towards the center of curvatureis reflected along the same ray. The emitted rays from the near-infraredlight emitting diodes are reflected off the corneal surface (e.g., firstPurkinje image) as glints, and are observed by the camera system. Theglints are extracted from the stereo image pair and used by the eyetracking module 220 to triangulate the corneal center 412 of each eye410 a and 410 b.

The distal light projectors 450 a and 450 b can emit visible lightray(s) 452 at display surface 330 in front of the eyes 410 a and 410 b.The emitted light ray(s) 452 is reflected off display surface 330towards the eye as reflected ray 454. The emitted light ray(s) 452 canbe adjusted by the active parallax correction module 230 to reflect offthe projection surface so that the reflected light ray 454 is coincidentwith the incident light ray 456 from visual content 320 to the cornealcenter 412.

It is contemplated that other eye tracking techniques and visual contentadjustment techniques can also be employed and are within the scope ofthe present disclosure. In some embodiments, one or more of the sameimage capture devices can be used to both capture image data of thereal-world visual content and perform eye tracking operations. In someembodiments, the image capture device(s) used for capturing image dataof the real-world visual content is/are different from the image capturedevice(s) used for performing the eye tracking operations disclosedherein. For example, one or more additional image capture devices (notshown) other than camera 460 a and 460 b can be employed to captureimage data of the real-world visual content.

FIGS. 5A-5B illustrate parameters that can be used in employing activeparallax correction in determining a display location of virtualcontent, in accordance with some embodiments. In FIG. 5A, the eye(s) 310of a user are looking at real-world visual content 320 through displaysurface 330 along a first gaze direction (e.g., angled to the left). Thefirst gaze direction intersects with display surface 330 at point ofintersection 510 a. In FIG. 5A, distance A represents the distancebetween the user's eye(s) and the point of intersection 510 a, whiledistance B represents the distance between the visual content 320 andthe point of intersection 510 a. The angle between the linesrepresenting distances A and B is represented as angle C. Similarly, inFIG. 5B, the eye(s) 310 of a user are looking at real-world visualcontent 320 through display surface 330 along a second gaze direction(e.g., angled to the right). The second gaze direction intersects withdisplay surface 330 at point of intersection 510 b. In FIG. 5B, distanceA represents the distance between the user's eye(s) and the point ofintersection 510 b, while distance B represents the distance between thevisual content 320 and the point of intersection 510 b. The anglebetween the lines representing distances A and B is represented as angleC. Distances A and B, as well as angle C can be used as parameters bythe active parallax correction module 230 in determining the location onthe display surface 330 at which to cause the virtual content (e.g.,virtual content 335 and 335′ in FIGS. 3A-3B) to be displayed. It iscontemplated that other parameters and configurations of parameters canbe employed in determining the location on the display surface 330 atwhich to cause the virtual content to be displayed.

FIG. 6 is a flowchart illustrating a method 600 of active parallaxcorrection, in accordance with some embodiments. The operations ofmethod 600 may be performed by a system or modules of a system (e.g.,virtual content module 150 in FIGS. 1-2). At operation 610, image dataof real-world visual content 320 can be received. The image data mayhave been captured by an image capture device. At operation 620, virtualcontent 335 to be displayed on a display surface 330 can be determinedbased on the received image data. At operation 630, a gaze direction ofat least one eye of a user can be determined, such as by using the eyetracking techniques disclosed herein or any other suitable eye trackingtechniques. At operation 640, a location on display surface 330 at whichto display the virtual content 335 can be determined. At operation 650,the virtual content 335 can be displayed on the display surface 330 atthe determined location. The method can repeat at operation 630, wherethe gaze direction of the user is once again determined and subsequentlyused in determining the location on the display surface 330 at which todisplay the virtual content 335. In this respect, the display locationof the virtual content 335 can be adjusted based on a detected change inthe gaze direction of the user. It is contemplated that the operationsof method 600 may incorporate any of the other features disclosedherein.

FIG. 7 is a flowchart illustrating a method 700 of active parallaxcorrection, in accordance with some embodiments. The operations ofmethod 700 may be performed by a system or modules of a system (e.g.,virtual content module 150 in FIGS. 1-2). At operation 710, one or morelight rays can be emitted from at least one light emitting diode towardsthe eye(s) of a user. At operation 720, at least one glint reflected offof the at least one eye can be observed (e.g., captured or detected)using at least one camera. At operation 730, a corneal center of the atleast one eye can be determined based on the observed glint(s). Atoperation 740, a path of incident light from the real-world visualcontent to the corneal center of the at least one eye can be determined.At operation 750, the positioning of at least one virtual content lightray forming the virtual content on the display surface can be adjustedto make at least one reflected light ray reflecting off of the displaysurface as a result of the virtual content light ray(s) coincident withthe incident light ray from the real-world visual content. It iscontemplated that the operations of method 700 may incorporate any ofthe other features disclosed herein.

In some embodiments, the determination of the gaze direction of the usercan be used to make other determinations about virtual content to bepresented to the user, in addition or as an alternative to thedetermination of the display location previously discussed. FIG. 8 is aflowchart illustrating a method 800 of using a gaze direction in thepresentation of virtual content to a user, in accordance with someembodiments. The operations of method 800 may be performed by a systemor modules of a system (e.g., virtual content module 150 in FIGS. 1-2).At operation 810, a gaze direction of at least one eye of a user can bedetermined, such as by using the eye tracking techniques disclosedherein or any other suitable eye tracking techniques.

At operation 820, a determination can be made about virtual content tobe presented to the user. In some embodiments, the determination caninclude a determination of a display location of the virtual content, aspreviously discussed. In some embodiments, the determination can includea selection of audio content for presentation to the user (e.g., playinga song file on the computing device 100 being used by the user). In someembodiments, the determination can include a selection of a program tolaunch. In some embodiments, the determination can include a selectionof a certain content progression to perform. In some embodiments, thedetermination can include a selection of haptic feedback to provide tothe user. Other types of determinations about the virtual content arealso within the scope of the present disclosure. In some embodiments,the determination at operation 820 can be made based on storedassociations between different gaze directions and different virtualcontent determinations. For example, a plurality of audio files can bestored in a database, with each audio file corresponding to a particulargaze direction. This mapping of gaze directions to audio files can thenbe used to select an audio file for presentation to the user based onthe determined gaze direction of the user. Other implementations andconfigurations are also within the scope of the present disclosure.

At operation 830, the virtual content can be presented to the user basedon the determination that was made at operation 820. It is contemplatedthat the operations of method 800 may incorporate any of the otherfeatures disclosed herein.

In some embodiments, a calibration process is performed on thecomponents of the computing device 100 prior to performance of theoperations, such as prior to the performance of the operations of method600 in FIG. 6, the operation of method 700 in FIG. 7, or the operationsof method 800 in FIG. 8. This calibration process may comprise thecalibration of any sensors, display surfaces, image capture devices, andprojectors of the computing device 100 so that they are allsynchronized, running off of the same clock, and knowing the attributes(e.g., location, positioning) of each other in order to facilitate theirworking together as a cohesive unit.

Modules, Components and Logic

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules. A hardware module is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain manner. In example embodiments, oneor more computer systems (e.g., a standalone, client, or server computersystem) or one or more hardware modules of a computer system (e.g., aprocessor or a group of processors) may be configured by software (e.g.,an application or application portion) as a hardware module thatoperates to perform certain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations. A hardware module may also compriseprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired) or temporarilyconfigured (e.g., programmed) to operate in a certain manner and/or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses) that connect the hardware modules. In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork (e.g., the network 214 of FIG. 2) and via one or moreappropriate interfaces (e.g., APIs).

Example embodiments may be implemented in digital electronic circuitry,or in computer hardware, firmware, software, or in combinations of them.Example embodiments may be implemented using a computer program product,e.g., a computer program tangibly embodied in an information carrier,e.g., in a machine-readable medium for execution by, or to control theoperation of, data processing apparatus, e.g., a programmable processor,a computer, or multiple computers.

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, subroutine,or other unit suitable for use in a computing environment. A computerprogram can be deployed to be executed on one computer or on multiplecomputers at one site or distributed across multiple sites andinterconnected by a communication network.

In example embodiments, operations may be performed by one or moreprogrammable processors executing a computer program to performfunctions by operating on input data and generating output. Methodoperations can also be performed by, and apparatus of exampleembodiments may be implemented as, special purpose logic circuitry(e.g., a FPGA or an ASIC).

A computing system can include clients and servers. A client and serverare generally remote from each other and typically interact through acommunication network. The relationship of client and server arises byvirtue of computer programs running on the respective computers andhaving a client-server relationship to each other. In embodimentsdeploying a programmable computing system, it will be appreciated thatboth hardware and software architectures merit consideration.Specifically, it will be appreciated that the choice of whether toimplement certain functionality in permanently configured hardware(e.g., an ASIC), in temporarily configured hardware (e.g., a combinationof software and a programmable processor), or a combination ofpermanently and temporarily configured hardware may be a design choice.Below are set out hardware (e.g., machine) and software architecturesthat may be deployed, in various example embodiments.

FIG. 9 is a block diagram of a machine in the example form of a computersystem 900 within which instructions 924 for causing the machine toperform any one or more of the methodologies discussed herein may beexecuted, in accordance with an example embodiment. In alternativeembodiments, the machine operates as a standalone device or may beconnected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in a server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a personal computer (PC), a tablet PC, a set-top box(STB), a Personal Digital Assistant (PDA), a cellular telephone, a webappliance, a network router, switch or bridge, or any machine capable ofexecuting instructions (sequential or otherwise) that specify actions tobe taken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein.

The example computer system 900 includes a processor 902 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 904 and a static memory 906, which communicate witheach other via a bus 908. The computer system 900 may further include avideo display unit 910 (e.g., a liquid crystal display (LCD) or acathode ray tube (CRT)). The computer system 900 also includes analphanumeric input device 912 (e.g., a keyboard), a user interface (UI)navigation (or cursor control) device 914 (e.g., a mouse), a disk driveunit 916, a signal generation device 918 (e.g., a speaker) and a networkinterface device 920.

The disk drive unit 916 includes a machine-readable medium 922 on whichis stored one or more sets of data structures and instructions 924(e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 924 mayalso reside, completely or at least partially, within the main memory904 and/or within the processor 902 during execution thereof by thecomputer system 900, the main memory 904 and the processor 902 alsoconstituting machine-readable media. The instructions 924 may alsoreside, completely or at least partially, within the static memory 906.

While the machine-readable medium 922 is shown in an example embodimentto be a single medium, the term “machine-readable medium” may include asingle medium or multiple media (e.g., a centralized or distributeddatabase, and/or associated caches and servers) that store the one ormore instructions 924 or data structures. The term “machine-readablemedium” shall also be taken to include any tangible medium that iscapable of storing, encoding or carrying instructions for execution bythe machine and that cause the machine to perform any one or more of themethodologies of the present embodiments, or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media. Specific examples of machine-readable mediainclude non-volatile memory, including by way of example semiconductormemory devices (e.g., Erasable Programmable Read-Only Memory (EPROM),Electrically Erasable Programmable Read-Only Memory (EEPROM), and flashmemory devices); magnetic disks such as internal hard disks andremovable disks; magneto-optical disks; and compact disc-read-onlymemory (CD-ROM) and digital versatile disc (or digital video disc)read-only memory (DVD-ROM) disks.

The instructions 924 may further be transmitted or received over acommunications network 926 using a transmission medium. The instructions924 may be transmitted using the network interface device 920 and anyone of a number of well-known transfer protocols (e.g., HTTP). Examplesof communication networks include a LAN, a WAN, the Internet, mobiletelephone networks, POTS networks, and wireless data networks (e.g.,WiFi and WiMax networks). The term “transmission medium” shall be takento include any intangible medium capable of storing, encoding, orcarrying instructions for execution by the machine, and includes digitalor analog communications signals or other intangible media to facilitatecommunication of such software.

Example Mobile Device

FIG. 10 is a block diagram illustrating a mobile device 1000 that mayemploy the active parallax correction features of the presentdisclosure, according to an example embodiment. The mobile device 1000may include a processor 1002. The processor 1002 may be any of a varietyof different types of commercially available processors 1002 suitablefor mobile devices 1000 (for example, an XScale architecturemicroprocessor, a microprocessor without interlocked pipeline stages(MIPS) architecture processor, or another type of processor 1002). Amemory 1004, such as a random access memory (RAM), a flash memory, orother type of memory, is typically accessible to the processor 1002. Thememory 1004 may be adapted to store an operating system (OS) 1006, aswell as application programs 1008, such as a mobile location enabledapplication that may provide LBSs to a user 102. The processor 1002 maybe coupled, either directly or via appropriate intermediary hardware, toa display 1010 and to one or more input/output (I/O) devices 1012, suchas a keypad, a touch panel sensor, a microphone, and the like.Similarly, in some embodiments, the processor 1002 may be coupled to atransceiver 1014 that interfaces with an antenna 1016. The transceiver1014 may be configured to both transmit and receive cellular networksignals, wireless data signals, or other types of signals via theantenna 1016, depending on the nature of the mobile device 1000.Further, in some configurations, a GPS receiver 1018 may also make useof the antenna 1016 to receive GPS signals.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader spirit and scope of the present disclosure. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense. The accompanying drawings that form a parthereof, show by way of illustration, and not of limitation, specificembodiments in which the subject matter may be practiced. Theembodiments illustrated are described in sufficient detail to enablethose skilled in the art to practice the teachings disclosed herein.Other embodiments may be utilized and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. This Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “invention” merelyfor convenience and without intending to voluntarily limit the scope ofthis application to any single invention or inventive concept if morethan one is in fact disclosed. Thus, although specific embodiments havebeen illustrated and described herein, it should be appreciated that anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

The Abstract of the Disclosure is provided to comply with 37 C.F.R.§1.72(b), requiring an abstract that will allow the reader to quicklyascertain the nature of the technical disclosure. It is submitted withthe understanding that it will not be used to interpret or limit thescope or meaning of the claims. In addition, in the foregoing DetailedDescription, it can be seen that various features are grouped togetherin a single embodiment for the purpose of streamlining the disclosure.This method of disclosure is not to be interpreted as reflecting anintention that the claimed embodiments require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle disclosed embodiment. Thus the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment

What is claimed is:
 1. A computer-implemented method comprising:determining a first gaze direction of at least one eye of a user;making, by a machine having a memory and at least one processor, adetermination about virtual content based on the first gaze direction;and causing the virtual content to be presented to the user based on thedetermination.
 2. The method of claim 1, wherein: making thedetermination comprises determining a first location on a displaysurface at which to display the virtual content, the first locationbeing determined based on the first gaze direction; and causing thevirtual content to be presented comprises causing the virtual content tobe displayed on the display surface at the first location.
 3. The methodof claim 2, further comprising: determining a second gaze direction ofthe at least one eye of the user, the second gaze direction beingdifferent from the first gaze direction; determining a second locationon the display surface at which to display the virtual content, thesecond location being determined based on the second gaze direction anddifferent from the first location; and causing the virtual content to bedisplayed on the display surface at the second location.
 4. The methodof claim 2, further comprising: receiving image data of real-worldvisual content, the image data having been captured by an image capturedevice; and determining the virtual content based on the received imagedata.
 5. The method of claim 4, wherein the determination of the firstlocation is further based on a distance between the at least one eye andthe display surface and a distance between the at least one eye and thereal-world visual content.
 6. The method of claim 4, wherein determiningthe first location comprises: emitting at least one light ray from atleast one light emitting diode towards the at least one eye; observingat least one glint reflected off of the at least one eye using at leastone camera; determining a corneal center of the at least one eye basedon the observed at least one glint; determining a path of incident lightfrom the real-world visual content to the corneal center of the at leastone eye; and adjusting positioning of at least one virtual content lightray forming the virtual content on the display surface to make at leastone reflected light ray reflecting off of the display surface as aresult of the at least one virtual content light ray coincident with theincident light ray from the real-world visual content.
 7. The method ofclaim 4, wherein the display surface is transparent.
 8. The method ofclaim 4, wherein the display surface and the image capture device arecoupled to a heads-up display device.
 9. The method of claim 8, whereinthe heads-up display device comprises an optical head-mounted displaydevice.
 10. A system comprising: a computing device having a memory andat least one processor; an eye tracking module configured to determine afirst gaze direction of at least one eye of a user; and a virtualcontent module, executable by the at least one processor, configured to:make a determination about virtual content based on the first gazedirection; and cause the virtual content to be presented to the userbased on the determination.
 11. The system of claim 10, wherein thevirtual content module comprises an active parallax correction moduleconfigured to: determine a first location on a display surface at whichto display the virtual content, the first location being determinedbased on the first gaze direction; and cause the virtual content to bedisplayed on the display surface at the first location.
 12. The systemof claim 11, wherein: the eye tracking module is further configured todetermine a second gaze direction of the at least one eye of the user,the second gaze direction being different from the first gaze direction;and the active parallax correction module is further configured to:determine a second location on the display surface at which to displaythe virtual content, the second location being determined based on thesecond gaze direction and different from the first location; and causethe virtual content to be displayed on the display surface at the secondlocation.
 13. The system of claim 11, further comprising: an imagecapture device configured to capture image data of real-world visualcontent; and a virtual content determination module configured todetermine the virtual content based on the received image data.
 14. Thesystem of claim 13, wherein the determination of the first location isfurther based on a distance between the at least one eye and the displaysurface and a distance between the at least one eye and the real-worldvisual content.
 15. The system of claim 13, further comprising: at leastone light emitting diode configured to emit at least one light raytowards the at least one eye; and at least one camera configured toobserve at least one glint reflected off of the at least one eye,wherein the eye tracking module is further configured to determine acorneal center of the at least one eye based on the observed at leastone glint, and wherein the active parallax correction module is furtherconfigured to determine a path of incident light from the real-worldvisual content to the corneal center of the at least one eye, and toadjust a positioning of at least one virtual content light ray formingthe virtual content on the display surface to make at least onereflected light ray reflecting off of the display surface as a result ofthe at least one virtual content light ray coincident with the incidentlight ray from the real-world visual content.
 16. The system of claim13, wherein the display surface is transparent.
 17. The system of claim13, wherein the display surface and the image capture device are coupledto a heads-up display device.
 18. The system of claim 17, wherein theheads-up display device comprises an optical head-mounted displaydevice.
 19. A non-transitory machine-readable storage medium, tangiblyembodying a set of instructions that, when executed by at least oneprocessor, causes the at least one processor to perform a set ofoperations comprising: determining a first gaze direction of at leastone eye of a user; making a determination about virtual content based onthe first gaze direction; and causing the virtual content to bepresented to the user based on the determination.
 20. The non-transitorymachine-readable storage medium of claim 19, wherein: making thedetermination comprises determining a first location on a displaysurface at which to display the virtual content, the first locationbeing determined based on the first gaze direction; and causing thevirtual content to be presented comprises causing the virtual content tobe displayed on the display surface at the first location.